We investigate the convergence rates and data sample sizes required for training a machine learning model using a stochastic gradient descent (SGD) algorithm, where data points are sampled based on either their loss value or uncertainty value. These training methods are particularly relevant for active learning and data subset selection problems. For SGD with a constant step size update, we present convergence results for linear classifiers and linearly separable datasets using squared hinge loss and similar training loss functions. Additionally, we extend our analysis to more general classifiers and datasets, considering a wide range of loss-based sampling strategies and smooth convex training loss functions. We propose a novel algorithm called Adaptive-Weight Sampling (AWS) that utilizes SGD with an adaptive step size that achieves stochastic Polyak's step size in expectation. We establish convergence rate results for AWS for smooth convex training loss functions. Our numerical experiments demonstrate the efficiency of AWS on various datasets by using either exact or estimated loss values.

Currently, Sharpness-Aware Minimization (SAM) is proposed to seek the parameters that lie in a flat region to improve the generalization when training neural networks. In particular, a minimax optimization objective is defined to find the maximum loss value centered on the weight, out of the purpose of simultaneously minimizing loss value and loss sharpness. For the sake of simplicity, SAM applies one-step gradient ascent to approximate the solution of the inner maximization. However, one-step gradient ascent may not be sufficient and multi-step gradient ascents will cause additional training costs. Based on this observation, we propose a novel random smoothing based SAM (R-SAM) algorithm. To be specific, R-SAM essentially smooths the loss landscape, based on which we are able to apply the one-step gradient ascent on the smoothed weights to improve the approximation of the inner maximization. Further, we evaluate our proposed R-SAM on CIFAR and ImageNet datasets. The experimental results illustrate that R-SAM can consistently improve the performance on ResNet and Vision Transformer (ViT) training.

Deep Neural Networks (DNNs) have yielded superior performance in many contemporary applications.

For samples where some annotators disagree, a comparative strategy is proposed to filter noise.

Currently, Sharpness-A ware Minimization (SAM) is proposed to seek the parameters that lie in a flat region to improve the generalization when training neural networks.

Statistics of the datasets used in our experiments.

A.1 PAC Bayesian Bound In this part, we provide a detailed P AC-Bound based on the continual learning scenario. We now consider the bound in the continual learning scenario. Based on Eq. (6), the expected error of C.1 Pseudo-code for FS-ER Comparing the pseudo-code of V anilla ER, FS-ER only adds the adversarial weight perturbation v . D.1 Datasets Table 4 summarizes the statistics of four datasets used in our experiments. Both fully connected layers have 2048 units in each layer.

Table S3 presents common loss functions and their hyperparameters.